{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 动态图与静态图\n",
    "\n",
    "<a href=\"https://gitee.com/mindspore/docs/blob/master/tutorials/source_zh_cn/middleclass/pynative_mode_and_graph_mode/pynative_mode_and_graph_mode.ipynb\" target=\"_blank\"><img src=\"https://gitee.com/mindspore/docs/raw/master/resource/_static/logo_source.png\"></a>\n",
    "\n",
    "## 概述\n",
    "\n",
    "MindSpore支持两种运行模式：\n",
    "\n",
    "- Graph模式：静态图模式或者图模式，将神经网络模型编译成一整张图，然后下发执行。该模式利用图优化等技术提高运行性能，同时有助于规模部署和跨平台运行。\n",
    "- PyNative模式：动态图模式，将神经网络中的各个算子逐一下发执行，方便用户编写和调试神经网络模型。\n",
    "\n",
    "默认情况下，MindSpore处于Graph模式，可以通过`context.set_context(mode=context.PYNATIVE_MODE)`切换为PyNative模式；同样地，MindSpore处于PyNative模式时，可以通过 `context.set_context(mode=context.GRAPH_MODE)`切换为Graph模式。\n",
    "\n",
    "Graph和PyNative两种模式的区别主要有：\n",
    "\n",
    "- 使用场景：Graph模式需要一开始就构建好网络结构，然后框架做整图优化和执行，比较适合网络固定没有变化，且需要高性能的场景。而PyNative模式逐行执行算子，支持单独求梯度。\n",
    "\n",
    "- 网络执行：Graph模式和PyNative模式在执行相同的网络和算子时，精度效果是一致的。由于Graph模式运用了图优化、计算图整图下沉等技术，Graph模式执行网络的性能和效率更高。\n",
    "\n",
    "- 代码调试：在脚本开发和网络流程调试中，推荐使用PyNative模式进行调试。在PyNative模式下，可以方便地设置断点，获取网络执行的中间结果，也可以通过pdb的方式对网络进行调试。而Graph模式无法设置断点，只能先指定算子进行打印，然后在网络执行完成后查看输出结果。\n",
    "\n",
    "下面以Graph模式为例，演示MindSpore单算子、普通函数、模型的执行方式，并进一步说明如何在PyNative模式下进行性能改进及梯度求取。\n",
    "\n",
    "## 执行方式\n",
    "\n",
    "这里演示在Graph模式和PyNative模式下，单算子、普通函数、模型的执行方式。\n",
    "\n",
    "> 在本案例的实际执行中，采取了MindSpore的默认方式`GRAPH_MODE`，用户也可以将其变更为`PYNATIVE_MODE`进行尝试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "from mindspore import context, Tensor, ParameterTuple, ms_function\n",
    "import mindspore.ops as ops\n",
    "from mindspore import dtype as mstype\n",
    "from mindspore.common.initializer import Normal\n",
    "from mindspore.nn import Dense, WithLossCell, SoftmaxCrossEntropyWithLogits, Momentum\n",
    "\n",
    "# 设定为Graph模式，也可替换为PYNATIVE_MODE\n",
    "context.set_context(mode=context.GRAPH_MODE, device_target=\"Ascend\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 执行单算子"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[[-0.02593261  0.01615404  0.01615404  0.01615404  0.01196378]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01535788  0.05602208  0.05602208  0.05602208  0.04094065]\n",
      "   [-0.01409336  0.04544117  0.04544117  0.04544117  0.0373004 ]]\n",
      "\n",
      "  [[ 0.03874376  0.02201786  0.02201786  0.02201786  0.02687691]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.05751193  0.02690699  0.02690699  0.02690699  0.03515062]\n",
      "   [ 0.02599058  0.01130002  0.01130002  0.01130002  0.02304572]]\n",
      "\n",
      "  [[-0.00022919  0.02640852  0.02640852  0.02640852  0.04932421]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.01657246  0.0705748   0.0705748   0.0705748   0.0874946 ]\n",
      "   [ 0.03821789  0.09614976  0.09614976  0.09614976  0.10491695]]\n",
      "\n",
      "  [[ 0.0190958   0.02602289  0.02602289  0.02602289  0.01660084]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.03556763  0.06862713  0.06862713  0.06862713  0.02653556]\n",
      "   [ 0.00727296  0.04514674  0.04514674  0.04514674  0.01423099]]]]\n"
     ]
    }
   ],
   "source": [
    "# 打印Cond2d算子的输出\n",
    "conv = nn.Conv2d(3, 4, 3, bias_init='zeros')\n",
    "input_data = Tensor(np.ones([1, 3, 5, 5]).astype(np.float32))\n",
    "output = conv(input_data)\n",
    "print(output.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 执行普通函数\n",
    "\n",
    "将若干算子组合成一个函数，然后直接通过函数调用的方式执行这些算子，并打印相关结果，如下例所示。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 4. 10. 18.]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import mindspore.nn as nn\n",
    "import mindspore.ops as ops\n",
    "from mindspore import context, Tensor\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.mul = ops.Mul()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        return self.mul(x, y)\n",
    "\n",
    "x = Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
    "\n",
    "net = Net()\n",
    "print(net(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 执行网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 4. 10. 18.]\n"
     ]
    }
   ],
   "source": [
    "class Net(nn.Cell):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.mul = ops.Mul()\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        return self.mul(x, y)\n",
    "\n",
    "x = Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32))\n",
    "y = Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32))\n",
    "\n",
    "net = Net()\n",
    "print(net(x, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## PyNative模式说明\n",
    "\n",
    "### 性能优化\n",
    "\n",
    "正如文章开头所说，Graph模式适合高性能的场景，但PyNative模式中也提供了性能优化的手段。MindSpore提供了Staging功能，该功能可以在PyNative模式下将Python函数或者Python类的方法编译成计算图，通过图优化等技术提高运行速度，是一种混合运行机制。Staging功能的使用通过`ms_function`装饰器达成，该装饰器会将将模块编译成计算图，在给定输入之后，以图的形式下发执行。如下例所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]\n",
      " [2. 2. 2. 2.]]\n"
     ]
    }
   ],
   "source": [
    "# 导入ms_function\n",
    "from mindspore import ms_function\n",
    "\n",
    "# 仍设定为PyNative模式\n",
    "context.set_context(mode=context.PYNATIVE_MODE, device_target=\"Ascend\")\n",
    "\n",
    "add = ops.Add()\n",
    "\n",
    "# 使用装饰器编译计算图\n",
    "@ms_function\n",
    "def add_fn(x, y):\n",
    "    res = add(x, y)\n",
    "    return res\n",
    "\n",
    "x = Tensor(np.ones([4, 4]).astype(np.float32))\n",
    "y = Tensor(np.ones([4, 4]).astype(np.float32))\n",
    "z = add_fn(x, y)\n",
    "print(z.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在加装了`ms_function`装饰器的函数中，如果包含不需要进行参数训练的算子（如`pooling`、`add`等算子），则这些算子可以在被装饰的函数中直接调用，如上例所示。如果被装饰的函数中包含了需要进行参数训练的算子（如`Convolution`、`BatchNorm`等算子），则这些算子必须在被装饰等函数之外完成实例化操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[[ 0.00994964  0.01850731 -0.05146599]\n",
      "   [ 0.02427048 -0.09082688 -0.00945184]\n",
      "   [ 0.02710651 -0.07322617  0.02594434]]\n",
      "\n",
      "  [[ 0.00056772 -0.05043615 -0.03873939]\n",
      "   [-0.00445028  0.03694705 -0.03555503]\n",
      "   [ 0.07329068 -0.02026664  0.01922888]]\n",
      "\n",
      "  [[ 0.02257145 -0.04093865 -0.00493869]\n",
      "   [ 0.01740007  0.02478302  0.02072578]\n",
      "   [ 0.05831327 -0.03933404  0.01767443]]\n",
      "\n",
      "  [[-0.03954437  0.02160874 -0.00700614]\n",
      "   [ 0.03856367 -0.04015685  0.02508826]\n",
      "   [-0.0229507  -0.03803677  0.02813173]]]\n",
      "\n",
      "\n",
      " [[[ 0.01678797 -0.02227589 -0.04470547]\n",
      "   [-0.05720481 -0.15464461  0.00911596]\n",
      "   [ 0.02566019 -0.04340314  0.03164666]]\n",
      "\n",
      "  [[ 0.03300299 -0.05849815  0.05841954]\n",
      "   [-0.11595733 -0.01524522  0.02947116]\n",
      "   [ 0.05930116  0.00831041 -0.0466827 ]]\n",
      "\n",
      "  [[-0.0797728   0.02910854  0.00766015]\n",
      "   [-0.01380327 -0.03338642  0.02625138]\n",
      "   [ 0.02279372 -0.00952736  0.02026749]]\n",
      "\n",
      "  [[ 0.04039776 -0.05340278 -0.0083563 ]\n",
      "   [ 0.04991922 -0.05205034 -0.0058607 ]\n",
      "   [ 0.00686666  0.00064385  0.00301326]]]]\n"
     ]
    }
   ],
   "source": [
    "# Conv2d实例化操作\n",
    "conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0)\n",
    "conv_obj.init_parameters_data()\n",
    "\n",
    "@ms_function\n",
    "def conv_fn(x):\n",
    "    res = conv_obj(x)\n",
    "    return res\n",
    "\n",
    "input_data = np.random.randn(2, 3, 6, 6).astype(np.float32)\n",
    "z = conv_fn(Tensor(input_data))\n",
    "print(z.asnumpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 梯度求取\n",
    "\n",
    "PyNative模式中支持单独的梯度求取操作，下面演示如何利用这一特性调试网络模型。具体操作中可通过`GradOperation`求该函数或者网络所有的输入梯度。需要注意，输入类型仅支持Tensor。\n",
    "\n",
    "构建网络如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"\n",
    "    Lenet网络结构\n",
    "    \"\"\"\n",
    "    def __init__(self, num_class=10, num_channel=1):\n",
    "        super(LeNet5, self).__init__()\n",
    "        # 定义所需要的运算\n",
    "        self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')\n",
    "        self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n",
    "        self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n",
    "        self.fc3 = nn.Dense(84, num_class, weight_init=Normal(0.02))\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "\n",
    "    def construct(self, x):\n",
    "        # 使用定义好的运算构建前向网络\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.max_pool2d(x)\n",
    "        x = self.flatten(x)\n",
    "        x = self.fc1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "# 实例化网络\n",
    "net = LeNet5()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上文所说，利用`GradOperation`求函数的输入梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GradWrap(nn.Cell):\n",
    "    \"\"\"求函数输入梯度\"\"\"\n",
    "    def __init__(self, network):\n",
    "        super(GradWrap, self).__init__(auto_prefix=False)\n",
    "        self.network = network\n",
    "        # 用Tuple的形式包装weight\n",
    "        self.weights = ParameterTuple(filter(lambda x: x.requires_grad, network.get_parameters()))\n",
    "\n",
    "    def construct(self, x, label):\n",
    "        weights = self.weights\n",
    "        # 返回值为梯度\n",
    "        return ops.GradOperation(get_by_list=True)(self.network, weights)(x, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在PyNative模式中进行网络训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.3025854\n"
     ]
    }
   ],
   "source": [
    "# 设定优化器、损失函数\n",
    "optimizer = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), 0.1, 0.9)\n",
    "criterion = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n",
    "\n",
    "# 通过WithLossCell获取Loss值\n",
    "net_with_criterion = WithLossCell(net, criterion)\n",
    "\n",
    "# 调用GradWrap\n",
    "train_network = GradWrap(net_with_criterion)\n",
    "train_network.set_train()\n",
    "\n",
    "# 产生输入数据\n",
    "input_data = Tensor(np.ones([32, 1, 32, 32]).astype(np.float32) * 0.01)\n",
    "label = Tensor(np.ones([32]).astype(np.int32))\n",
    "output = net(Tensor(input_data))\n",
    "\n",
    "# 利用前向网络计算loss\n",
    "loss_output = criterion(output, label)\n",
    "# 求得梯度\n",
    "grads = train_network(input_data, label)\n",
    "# 优化参数\n",
    "success = optimizer(grads)\n",
    "loss = loss_output.asnumpy()\n",
    "print(loss)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore1.2_gpu",
   "language": "python",
   "name": "liuxiao_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}